Information processing device, image generation method, and head-mounted display

ABSTRACT

A motion detection section  30  detects the posture of an HMD worn on the head of a user. A status determination section  32  determines a user&#39;s gaze direction and a binocular inclination angle in accordance with the detected posture of the HMD. The binocular inclination angle is the angle between a horizontal plane and a line connecting the left and right eyes of the user. An image identification section  34  determines two images for use in the generation of left- and right-eye parallax images from a plurality of viewpoint images stored in an image storage device  16  in accordance with the user&#39;s gaze direction and the binocular inclination angle. An image generation section  36  generates left- and right-eye parallax images from the two identified images. An image supply section  38  supplies the generated parallax images to the HMD.

TECHNICAL FIELD

The present invention relates to a technology for generating parallaximages to be displayed on a head-mounted display and/or a technology fordisplaying parallax images on a head-mounted display.

BACKGROUND ART

The head-mounted display (HMD) is worn on the head of a user in order toprovide a virtual reality (VR) world for the user. The sense ofimmersion into a video world can be enhanced by incorporating a headtracking function into the HMD and updating a display screen inconjunction with the motion of the user's head. In recent years, atechnology for taking a panoramic photograph is widespread. Displaying acaptured panoramic photograph image on the HMD gives the user the senseof being in a real-world place.

CITATION LIST Patent Literature PTL 1

JP 2015-95045A

SUMMARY Technical Problems

The HMD includes a display panel for a user's left eye and a displaypanel for a user's right eye. The display panels respectively display aleft-eye image and a right-eye image (parallax images) to providestereoscopic vision. As regards a game application, a left-eye virtualcamera and a right-eye virtual camera can be disposed in a game space togenerate left-eye and right-eye game images and output the generatedgame images from the respective display panels of the HMD.

As regards a VR application that displays an actually captured objectimage on the HMD, a content image used as a material is generallycaptured by stereo cameras for displaying stereoscopic vision. When thestereo cameras are used to capture images of an object, the images arecaptured each time the left camera and the right camera, which aremaintained in a horizontal position, are rotationally transferred by apredetermined amount in up-down direction and left-right direction froma fixed point. Therefore, when the user wearing the HMD shakes his/herhead vertically or horizontally, the left and right eyes of the user areat the same horizontal height. Consequently, parallax images having afixed parallax amount can be generated based on a stereo image material.

However, in a case where the user facing forward tilts his/her headlaterally (tilts the head toward a shoulder), a line connecting the leftand right eyes is inclined from a horizontal plane. Thus, the left andright eyes are not at the same horizontal height. In this case, if nomaterial is imaged with the stereo cameras tilted at the same angle, itis difficult to generate parallax images having a fixed parallax amount.In reality, it is almost impossible to capture images of an object withthe stereo cameras tilted at all angles from the horizontal plane.Therefore, if the user tilts his/her head laterally during the use ofthe HMD, it is impossible to provide appropriate parallax images. Undersuch circumstances, desired is the development of a technology forgenerating parallax images having a fixed parallax amount even when theuser wearing the HMD tilts his/her head laterally.

The present invention has been made in view of the above circumstances.An object of the present invention is to provide a technology forproperly generating parallax images to be displayed on the HMD and/or atechnology for displaying parallax images.

Solution to Problems

In accomplishing the above objects, according to an aspect of thepresent invention, there is provided an information processing deviceincluding a detection section, a status determination section, an imageidentification section, an image generation section, and an image supplysection. The detection section detects the posture of a head-mounteddisplay worn on the head of a user. The status determination sectiondetermines a user's gaze direction and a binocular inclination angle inaccordance with the posture of the head-mounted display, which isdetected by the detection section. The binocular inclination angle isthe angle between a horizontal plane and a line connecting the left andright eyes. The image identification section identifies two images usedto generate left- and right-eye parallax images from a plurality ofviewpoint images in accordance with the user's gaze direction andbinocular inclination angle determined by the status determinationsection. The image generation section generates left- and right-eyeparallax images from the two images identified by the imageidentification section. The image supply section supplies the parallaximages generated by the image generation section to the head-mounteddisplay.

According to another aspect of the present invention, there is providedthe information processing device including a detection section, astatus determination section, an image identification section, an imagegeneration section, and an image supply section. The detection sectiondetects the posture of a head-mounted display worn on the head of auser. The status determination section determines a user's gazedirection and a binocular inclination angle in accordance with theposture of the head-mounted display, which is detected by the detectionsection. The binocular inclination angle is the angle between ahorizontal plane and a line connecting the left and right eyes. Theimage identification section identifies at least two images used togenerate left- and right-eye parallax images from a plurality ofviewpoint images in accordance with the user's gaze direction andbinocular inclination angle determined by the status determinationsection. The plurality of viewpoint images include a plurality ofviewpoint images acquired in a first gaze direction and/or a pluralityof viewpoint images acquired in a second gaze direction. The imagegeneration section generates left- and right-eye parallax images from atleast two images identified by the image identification section. Theimage supply section supplies the parallax images generated by the imagegeneration section to the head-mounted display.

According to still another aspect of the present invention, there isprovided a head-mounted display including a left-eye display panel and aright-eye display panel. The head-mounted display further includes acontrol section that receives left- and right-eye image data anddisplays the received image data on the left-eye display panel and theright-eye display panel. The left-and right-eye image data are generatedfrom two viewpoint images that are identified based on the gazedirection of a user wearing the head-mounted display and on a binocularinclination angle. The binocular inclination angle is the angle betweena horizontal plane and a line connecting the left and right eyes of theuser.

A combination of the above-mentioned elements and an expression of thepresent invention that are converted between, for example, methods,devices, systems, computer programs, recording media storing readablecomputer programs, and data structures are also effective as an aspectof the present invention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an exemplary configuration of aninformation processing system according to an embodiment.

FIG. 2 is a diagram illustrating an exemplary configuration of amulti-viewpoint camera.

FIGS. 3(a) to 3(c) are diagrams illustrating viewpoint images capturedby individual cameras of the multi-viewpoint camera.

FIGS. 3(d) and 3(e) are diagrams illustrating viewpoint images capturedby individual cameras of the multi-viewpoint camera.

FIGS. 4(a) and 4(b) are diagrams illustrating a state where viewpointimages are disposed in a virtual two-dimensional coordinate plane.

FIG. 5 is a diagram illustrating exemplary coordinate values ofviewpoint positions associated with viewpoint images.

FIG. 6 is a diagram illustrating parallax images.

FIG. 7 is a diagram illustrating an exemplary external form of an HMD.

FIG. 8 is a diagram illustrating the functional blocks of the HMD.

FIG. 9 is a diagram illustrating the functional blocks of an informationprocessing device.

FIGS. 10(a) and 10(b) are diagrams illustrating a binocular inclinationangle.

FIG. 11 is a diagram illustrating an intersection between a user's gazedirection and a two-dimensional coordinate plane.

FIG. 12 is a diagram illustrating an example in which the position ofthe user's gaze direction in the two-dimensional coordinate plane isderived.

FIGS. 13(a) and 13(b) are diagrams illustrating examples of left- andright-eye image positions.

FIG. 14 is a diagram illustrating the gaze direction and angle-of-viewof multi-viewpoint images.

FIG. 15 is a diagram illustrating the relationship betweentwo-dimensional coordinate planes of multi-viewpoint images.

FIG. 16 is a diagram illustrating an exemplary user's gaze direction.

FIG. 17 is a diagram illustrating an example in which parallax imagesare generated by using both a first image group and a second imagegroup.

DESCRIPTION OF EMBODIMENTS

FIG. 1 illustrates an exemplary configuration of an informationprocessing system 1 according to an embodiment. The informationprocessing system 1 includes an information processing device 10, ahead-mounted display device (HMD) 100, an input device 6, an imagingdevice 7, and an output device 3. The HMD 100 is to be worn on the headof a user. The input device 6 is to be operated by user's fingers. Theimaging device 7 captures an image of the user wearing the HMD 100. Theoutput device 4 displays an image.

The information processing device 10 according to the embodimentincludes a processing device 12, an output control device 14, and animage storage device 16. The processing device 12 is a terminal devicethat receives operation information from the user-operated input device6 and executes various applications. The processing device 12 may beconnected to the input device 6 by a cable or by a known wirelesscommunication protocol. The output control device 14 is a processingunit that outputs image data generated by the processing device 12 tothe HMD 100. The output control device 14 may be connected to the HD 100by a cable or by a well-known wireless communication protocol.

The imaging device 7 is a stereo camera that captures an image of theuser wearing the HMD 100 at predetermined intervals and supplies thecaptured image to the processing device 12. Although described in detaillater, the HMD 100 is provided with markers (tracking LEDs) for trackingthe head of the user, and the processing device 12 detects the motion ofthe HMD 100 in accordance with the position of the marker included inthe captured image. It should be noted that posture sensors(acceleration sensor and gyro sensor) are mounted in the HMD 100. Theprocessing device 12 performs a high-precision tracking process byacquiring sensor information detected by the posture sensors from theHMD 100 and using the acquired sensor information together with thecaptured image showing the marker. It should be noted that variousmethods have already been proposed for the tracking process. Theprocessing device 12 may adopt any tracking method as far as it detectsthe motion of the HMD 100.

The information processing system 1 according to the embodiment executesa virtual-reality (VR) application that supplies parallax images to theHMD 100. As the user wears the HMD 100, the output device 4 is notalways required for the information processing system 1. However, theoutput control device 14 or the processing device 12 may cause theoutput device 4 to output the same image as the image to be displayed onthe HMD 100. Consequently, the output device 4 allows another user toview an image that the user is viewing on the HMD 100. It should benoted that images outputted from the output device 4 need not always beparallax images although the HMD 100 outputs parallax images.

The HMD 100 is a display device that is to be worn on the head of theuser and is adapted to display images on display panels disposed beforeuser's eyes. The HMD 100 displays an image for a left-eye on a left-eyedisplay panel and displays an image for a right-eye on a right-eyedisplay panel. The displayed images are parallax images as viewed fromleft and right viewpoints and adapted to provide a stereoscopic view. Itshould be noted that the user views the display panels through anoptical lens. Therefore, the information processing device 10 obtainsparallax image data by correcting optical distortion of the opticallens, and supplies the obtained parallax image data to the HMD 100. Thisoptical distortion correction process may be performed by either theprocessing device 12 or the output control device 14.

In the information processing system 1, the processing device 12, theoutput device 4, the input device 6, and the imaging device 7 mayconstruct a conventional game system. In such a case, the processingdevice 12 is a gaming device that executes a game, and the input device6 supplies user's operation information to the processing device 12,such as a game controller, a keyboard, a mouse, or a joystick. Addingthe output control device 14, the image storage device 16, and the HMD100 to the component elements of the game system constructs theinformation processing system 1 that executes a VR application.

It should be noted that the functions of the output control device 14may be incorporated into the processing device 12 as part of thefunctions of the VR application. That is to say, a processing unit ofthe information processing device 10 may merely include the processingdevice 12 or include the processing device 12 and the output controldevice 14. Subsequently, the functions of the processing device 12 andoutput control device 14 that are required for executing the VRapplication will be collectively described as the functions of theinformation processing device 10.

As a prerequisite for executing the VR application, a plurality ofviewpoint images acquired in the same gaze direction from a plurality ofviewpoint positions are stored in the image storage device 16. Theinformation processing device 10 generates left- and right-eye parallaximages by using at least two viewpoint images stored in the imagestorage device 16, and supplies the generated parallax images to the HMD100.

FIG. 2 illustrates an exemplary configuration of a multi-viewpointcamera 18. The multi-viewpoint camera 18 captures viewpoint images thatare used as the materials for parallax images. The multi-viewpointcamera 18 includes a plurality of cameras 19 a-19 i that are arrayed ina matrix format in the same plane. In the exemplary configurationillustrated in FIG. 2, the cameras 19 a-19 i are arrayed at equalintervals in a three-row by three-column format. However, the cameras 19a-19 i may be arrayed in a different format. When the cameras 19 a-19 iare not distinguished from each other, they may be generically referredto as the “cameras 19.” The cameras 19 each include a solid-state imagesensor such as a CMOS. (Complementary Metal Oxide Semiconductor) sensoror a CCD (Charge-Coupled Device) sensor.

The optical axis (gaze direction) of each camera 19 is set to be adirection perpendicular to the front of the multi-viewpoint camera 18.That is to say, the optical axes of the cameras 19 are set to beparallel to each other. Therefore, two adjacent cameras 19 captureimages of an object in gaze directions parallel to each other so thatthe captured images are displaced from each other by an inter-cameradistance (baseline length). As the cameras 19 simultaneously capture theimages of the object, the multi-viewpoint camera 18 acquires a pluralityof viewpoint images based on the viewpoint positions of the cameras 19.

The following describes the positional relationship between viewpointimages that are obtained by capturing images of a certain scene with themulti-viewpoint camera 18. FIGS. 3(a) to 3(e) depict viewpoint images 17captured by the cameras 19. The left-right positions of the cameras 19are defined based on a direction from the cameras 19 toward an object(i.e., the left-right positions depicted in FIG. 2 are reversed).

FIG. 3(a) depicts a viewpoint image 17 e that is captured by a camera 19e positioned at the center of the multi-viewpoint camera 18.

FIG. 3(b) depicts a viewpoint image 17 c and a viewpoint image 17 g. Theviewpoint image 17 c is captured by an upper left camera 19 c. Theviewpoint image 17 g is captured by a lower right camera 19 g.

FIG. 3(c) depicts a viewpoint image 17 a and a viewpoint image 17 i. Theviewpoint image 17 a is captured by an upper right camera 19 a. Theviewpoint image 17 i is captured by a lower left camera 19 i.

FIG. 3(d) depicts a viewpoint image 17 b and a viewpoint image 17 f. Theviewpoint image 17 b is captured by an upper central camera 19 b. Theviewpoint image 17 f is captured by a left central camera 19 f.

FIG. 3(e) depicts a viewpoint image 17 h and a viewpoint image 17 d. Theviewpoint image 17 h is captured by a lower central camera 19 h. Theviewpoint image 17 d is captured by a right central camera 19 d.

The multi-viewpoint camera 18 simultaneously acquires a plurality ofviewpoint images 17 a-17 i. The distance between adjacent cameras 19arrayed vertically or horizontally is set, for example, to 20 mm. As theoptical axes of the cameras 19 are parallel to each other, imagescaptured by different cameras 19 are as viewed in the same imagingdirection (gaze direction) from different viewpoint positions.Therefore, two images captured by vertically and horizontally adjacentcameras 19 have a parallax of 20 mm.

In the embodiment, interpolated images obtained by interpolating aplurality of viewpoint images 17 captured by the multi-viewpoint camera18 is generated in advance so that parallax images supplied to the usersmoothly move in conjunction with the motion of the HMD 100 worn by theuser. The interpolated images correspond to the viewpoint images thatare captured from different viewpoint positions in the same gazedirection as the gaze direction of the multi-viewpoint camera 18. If theviewpoint images 17 captured by the actual cameras 19 are called “actualviewpoint images,” the interpolated images generated by an interpolationprocess performed on at least two actual viewpoint images can be called“virtual viewpoint images.” The interpolation process on the actualviewpoint images may be performed by the information processing device10. However, such an interpolation process may alternatively beperformed, for example, by an image server supplying a content image,and supplied to the information processing device 10.

FIG. 4(a) illustrates a state where actual viewpoint images captured bythe multi-viewpoint camera 18 are disposed in a virtual two-dimensionalplane (two-dimensional coordinate plane). Two-dimensional coordinatesindicate camera positions (viewpoint positions) at which images of anobject are captured. In the embodiment, the distance (baseline length)between vertically and horizontally adjacent cameras 19 is 20 mm.Therefore, the two-dimensional coordinate plane depicted in FIG. 4(a)also indicates that the amount of parallax between vertically andhorizontally adjacent viewpoint images 17 is 20 mm.

FIG. 4(b) illustrates a state where interpolated images (virtualviewpoint images) interpolating a plurality of actual viewpoint imagesare disposed in a virtual two-dimensional coordinate plane. Here, aplurality of virtual viewpoint images are generated between a pluralityof viewpoint images 17 captured by the cameras 19 in such a manner thatthe amount of parallax between adjacent viewpoint images is 2 mm. FIG.4(b) indicates that the interpolated images represented by thin linesare generated between the viewpoint images 17 represented by thicklines. Various methods for generating interpolated images have beenproposed. The processing device 12 may generate the interpolated imagesby using a well-known algorithm. An exemplary interpolation process isdescribed below.

First of all, based on a plurality of actual viewpoint images, theprocessing device 12 acquires depth information regarding an image of anobject. The depth information is acquired, for example, by performing astereo matching process. A difference (parallax) based on the distanceto the object exists between two-dimensional images captured from twoviewpoints. Stereo matching is a method of determining three-dimensionaldepth information by using the above-mentioned difference. Based on aplurality of actual viewpoint images and the depth information, theprocessing device 12 generates a plurality of interpolated imagesbetween the actual viewpoint images, and stores the generatedinterpolated images in the image storage device 16.

Interpolated images in an upper right area surrounded by viewpointimages 17 a, 17 b, 17 e, and 17 d will now be described. Nine virtualviewpoint images acquired by laterally moving the viewpoint position 2mm at a time are generated between the viewpoint image 17 a and theviewpoint image 17 b. Further, nine virtual viewpoint images acquired bylaterally moving the viewpoint position 2 mm at a time are generatedbetween the viewpoint image 17 d and the viewpoint image 17 e. As aresult, eleven viewpoint images are lined up beginning with theviewpoint image 17 a and ending with the viewpoint image 17 b, andeleven viewpoint images are lined up beginning with the viewpoint image17 d and ending with the viewpoint image 17 e. The viewpoint position isthen vertically moved 2 mm at a time between the above-mentionedviewpoint images to acquire virtual viewpoint images. In this manner,nine virtual viewpoint images are generated for each column. As aresult, multi-viewpoint images are formed in the upper right area insuch a manner that the amount of parallax between vertically andhorizontally adjacent viewpoint images is 2 mm. When the similarinterpolation process is performed on the other areas, a group of 441(21 columns by 21 rows) viewpoint images is generated.

In the image storage device 16, each viewpoint image is stored inassociation with its viewpoint position. The viewpoint positions are setas coordinate values of virtual two-dimensional coordinates, and thecoordinate values are used to calculate an actual distance.

FIG. 5 illustrates exemplary coordinate values of viewpoint positionsassociated with viewpoint images. In the two-dimensional coordinateplane depicted in FIG. 5, for the sake of convenience, the viewpointposition for the viewpoint image 17 e is set as the origin (0,0). Inthis instance, the viewpoint position for the viewpoint image 17 a isexpressed as (10,10), and the viewpoint position for the viewpoint image17 h is expressed as (0, −10). The similar holds for interpolatedviewpoint images 17. The viewpoint position for the viewpoint image 17 mis expressed as (6, −4), and the viewpoint position for the viewpointimage 17 n is expressed as (−5,5). In the above-mentionedtwo-dimensional coordinates, one division (one coordinate value) of thex- and y-axes corresponds to 2 mm. Therefore, if two viewpoint positionsare (x1,y1) and (x2,y2), respectively, the distance between the twoviewpoint positions is expressed by Equation (1) below.√{square root over ((x1−x2)²+(y1−y2)²)}×2 mm  [Math1]

It should be noted that the above method of expressing viewpointpositions in the two-dimensional coordinate plane is merely an example.The viewpoint positions may be managed in any manner as far as thedistance between two viewpoint positions can be calculated.

Multi-viewpoint images whose amount of parallax between vertically andhorizontally adjacent viewpoint images is 2 mm may be generated byperforming an interpolation process. However, such multi-viewpointimages may alternatively be acquired by actually capturing images. Forexample, a single camera 19 may be moved 2 mm at a time in the same gazedirection within the range of 40 mm vertically and 40 mm horizontally,and an image capturing operation may be performed each time the camerais moved to acquire a total of 441 viewpoint images whose amount ofparallax between adjacent viewpoint images is 2 mm. Further, a pluralityof multi-viewpoint images may be acquired in such a manner that theacquired images are spatially contiguous in a transverse direction, andthen stored in the storage device 16. A left-right, all-around panoramicimage is formed by joining different multi-viewpoint images.

Based on a user's gaze direction determined from the posture of the HMD100 worn on the head of the user and a binocular inclination that is theangle between a horizontal plane and a line connecting the left andright eyes of the user, the information processing device 10 accordingto the embodiment causes the HMD 100 to display appropriate parallaximages. Further, the information processing device 10 may generateparallax images based on a user's viewpoint position determined from theposition of the HMD 100.

FIG. 6 is a diagram illustrating parallax images generated in theinformation processing device 10. The VR application according to theembodiment assumes that the image storage device 16 storesmulti-viewpoint images acquired with a direction (horizontal direction)parallel to the horizontal plane set as the imaging direction (gazedirection), and that the user views, through the HMD 100, the parallaximages in a gaze direction parallel to the horizontal plane. It shouldbe noted that the above conditions are merely mentioned to facilitateunderstanding. As described later, the gaze direction of themulti-viewpoint camera 18 for acquiring multi-viewpoint images and thegaze direction of the user wearing the HMD 100 are not limited to thehorizontal direction.

The image storage device 16 stores laterally contiguous differentmulti-viewpoint images in association with their respective gazedirections. Here, the term “contiguous” signifies that all existingobjects are imaged from a 360-degree circumference. In the embodiment,the different multi-viewpoint images are generated based on a pluralityof viewpoint images that are captured each time the angle of themulti-viewpoint camera 18 from the horizontal plane is shifted by apredetermined angle of β around a predetermined position. The VRapplication according to the embodiment enables the user to enjoycontent images around himself/herself when the user moves his/her facewhile facing towards the horizontal direction.

When a user's face moves to change the gaze direction and the binocularinclination angle, which is the angle between the horizontal plane andthe line connecting the left and right eyes of the user, the gazedirection of a virtual camera 8 and the tilt of the left and rightcameras are changed to update a left-eye image 5 a and a right-eye image5 b that are supplied to the HMD 100.

As indicated in FIG. 6, viewpoint images serving as image materials maybe attached to the inner circumferential surface of a virtualcylindrical body. When a plurality of viewpoint images are virtuallydisposed on the inner surface of the cylindrical body in a contiguousmanner (with no gaps), a panoramic image in the VR application isformed. Consequently, a more real video world can be provided to theuser.

The information processing device 10 detects the position coordinatesand posture of a user's head (actually the HMD 100) by performing a headtracking process on the user. Here, the position coordinates of the HMD100 are position coordinates in a three-dimensional space where areference position is used as the origin. Position coordinates achievedwhen the HMD 100 is turned on may be set as the reference position.Further, the posture of the HMD 100 is the inclination of a three-axisdirection from a reference posture in the three-dimensional space. Thereference posture is such that the binocular inclination angle, which isthe angle between the horizontal plane and the line connecting the leftand right eyes, is zero (i.e., the left and right eyes are at the sameheight), and that the user's gaze direction is the horizontal direction.The reference posture may be set when the HMD 100 is turned on.

A well-known technology may be used for the head tracking process. Theinformation processing device 10 is able to detect the positioncoordinates and posture of the HMD 100 from only the sensor informationdetected by the posture sensors of the HMD 100, and is further able todetect the position coordinates and posture of the HMD 100 with highprecision by performing an image analysis process on the markers(tracking LEDs) of the HMD 100, which are imaged by the imaging device7.

In accordance with the detected position coordinates and posture of theHMD 100, the information processing device 10 determines the positionand posture of the virtual camera 8 in the virtual cylindrical body. Thevirtual camera 8 is a stereo camera and disposed to capture an image ofthe inner circumferential surface of the virtual cylindrical body fromthe inside of the virtual cylindrical body. It should be noted that theposition coordinates of the HMD 100 specify the position of the virtualcamera 8 in the virtual cylindrical body, and that the posture of theHMD 100 specifies the posture of the virtual camera 8 in the virtualcylindrical body.

The relationship between the posture of the HMD 100 and the posture ofthe virtual camera 8 will now be described. The information processingdevice 10 causes the posture of the virtual camera 8 to coincide withthe detected posture of the HMD 100. The left camera in the virtualcamera 8 virtually captures the left-eye image 5 a, and the right camerain the virtual camera 8 virtually captures the right-eye image 5 b. Theleft-eye image 5 a and the right-eye image 5 b are displaced from eachother by the distance (baseline length) between the right camera and theleft camera. The amount of this displacement forms a parallax.

The baseline length in the virtual camera 8 determines the amount ofparallax between the left-eye image 5 a and the right-eye image 5 b.Therefore, it is preferable that the baseline length in the virtualcamera 8 be user-changeable as needed. The amount of parallax needs tobe increased to produce an enhanced stereoscopic effect. However, anincrease in the amount of parallax may give some users an uncomfortablefeeling or viewing fatigue. It is generally said that the distancebetween the left and right eyes of a human adult is approximately 65 mm.However, if the amount of parallax between the left-eye image 5 a andthe right-eye image 5 b is set to 65 mm in the HMD 100 for experimentpurposes, it is found that images are extremely difficult to view. Inview of such circumstances, the embodiment sets the baseline length ofthe virtual camera 8 to 20 mm by default, and allows the user to changethe baseline length of the virtual camera 8 to a desired value before orafter the start of the VR application.

FIG. 7 illustrates an exemplary external form of the HMD 100. The HMD100 includes an output mechanism section 102 and a mounting mechanismsection 104. The mounting mechanism section 104 includes a mounting band106 that fastens the HMD 100 to the whole circumference of the user'shead when the user wears the HMD 100. The mounting band 106 has amaterial or structure that enables the user to adjust the length of themounting band 106 until it fits on the circumference of the user's head.

The output mechanism section 102 includes a housing 108 and a displaypanel. The housing 108 is shaped so as to cover the left and right eyesof the user when the user wears the HMD 100. The display panel isdisposed inside the housing 108 and adapted to face the eyes of the userwhen the user wears the HMD 100. The display panel may be, for example,a liquid-crystal panel or an organic EL panel. A left and right pair ofoptical lenses are further disposed in the housing 108. The opticallenses are positioned between the display panel and the user's eyes toincrease the viewing angle of the user. The HMD 100 may further includespeakers and earphones that are positioned to match user's ears.

Light-emitting markers 110 a, 110 b, 110 c, 110 d are attached to theouter surface of the housing 108. In the present example, the trackingLEDs are used as the light-emitting markers 110. However, differenttypes of markers may alternatively be used. Any markers may be used asfar as they can be imaged by the imaging device 7 and their positionscan be subjected to image analysis by the information processing device10. The number of light-emitting markers 110 and their positions are notparticularly limited. However, an adequate number of light-emittingmarkers 110 need to be properly disposed to detect the posture of theHMD 100. In the example depicted in FIG. 7, the light-emitting markers110 are disposed at four front corners of the housing 108. Further, thelight-emitting markers 110 may be disposed on the sides and rear of themounting band 106 in order to capture an image even when the back of theuser faces the imaging device 7.

The HMD 100 may be connected to the information processing device 10 bya cable or by a well-known wireless communication protocol. The HMD 100transmits sensor information detected by the posture sensors to theinformation processing device 10, and receives the parallax image datagenerated by the information processing device 10 in order to displaythe parallax image data on the left-eye display panel and the right-eyedisplay panel.

FIG. 8 illustrates the functional blocks of the HMD 100. A controlsection 120 is a main processor that processes commands and variousdata, such as image data, audio data, and sensor information, andoutputs the result of processing. A storage section 122 temporarilystores, for example, data and commands to be processed by the controlsection 120. A posture sensor 124 detects posture information regardingthe HMD 100. The posture sensor 124 includes at least a three-axisacceleration sensor and a three-axis gyro sensor.

For the HMD 100, XYZ three-dimensional coordinates are set as depictedin FIG. 7. Here, a pitch axis, a yaw axis, and a roll axis are set asthe X-axis, the Y-axis, and the Z-axis, respectively. A firstacceleration sensor detects the acceleration in the pitch axisdirection, and a first gyro sensor detects the angular velocity aroundthe pitch axis. A second acceleration sensor detects the acceleration inthe yaw axis direction, and a second gyro sensor detects the angularvelocity around the yaw axis. A third acceleration sensor detects theacceleration in the roll axis direction, and a third gyro sensor detectsthe angular velocity around the roll axis.

A communication control section 128 establishes wired or wirelesscommunication through a network adapter or an antenna, and transmitsdata outputted from the control section 120 to the external informationprocessing device 10. Further, the communication control section 128establishes wired or wireless communication through the network adapteror the antenna, receives data from the information processing device 10,and outputs the received data to the control section 120.

Upon receiving video data and audio data from the information processingdevice 10, the control section 120 supplies the received video data to adisplay panel 130 for the purpose of display, and supplies the receivedaudio data to an audio output section 132 for the purpose of audiooutput. The display panel 130 includes a left-eye display panel 130 aand a right-eye display panel 130 b. The left- and right-eye displaypanels display a pair of parallax images having a predetermined parallaxamount. Further, the control section 120 causes the communicationcontrol section 128 to transmit sensor information received from theposture sensor 124 and audio data received from a microphone 126 to theinformation processing section 10.

FIG. 9 illustrates the functional blocks of the information processingdevice 10. As input/output interfaces to the outside, the informationprocessing device 10 includes a sensor information acquisition section20, a captured image acquisition section 22, a reception section 24, andan image supply section 38. The sensor information acquisition section20 acquires sensor information at predetermined intervals from theposture sensor 124 of the HMD 100. The captured image acquisitionsection 22 acquires a captured image of the HMD 100 at predeterminedintervals from the imaging device 7. For example, the imaging device 7captures an image of a forward space at 1/60-second intervals, and thecaptured image acquisition section 22 acquires a captured image at1/60-second intervals. The reception section 24 acquires user-inputtedinformation from the input device 6.

The information processing device 10 further includes a motion detectionsection 30, a status determination section 32, an image identificationsection 34, and an image generation section 36. The motion detectionsection 30 detects the position coordinates and posture of the HMD 100in accordance with the motion of the HMD 100 worn on the head of theuser. The status determination section 32 determines the user'sviewpoint position in accordance with the position coordinates of theHMD 100, which is detected by the motion detection section 30, anddetermines the user's gaze direction and the binocular inclinationangle, which is the angle between the horizontal plane and the lineconnecting the left and right eyes of the user, in accordance with theposture of the HMD 100.

From a plurality of parallax images stored in the image storage device16, the image identification section 34 identifies two images for use inthe generation of left-and right-eye parallax images. In this instance,based on at least the user's gaze direction and binocular inclinationangle determined by the status determination section 32, the imageidentification section 34 identifies the two images for use in thegeneration of the parallax images. For the sake of convenience, theembodiment assumes that the user's gaze direction is parallel to thehorizontal plane. The image generation section 36 generates the left-and right-eye parallax images from the two images identified by theimage identification section 34. The image supply section 38 suppliesthe parallax images generated by the image generation section 36 to theHMD 100.

Referring to FIG. 9, individual elements depicted as the functionalblocks for performing various processes may be formed by hardware, suchas a circuit block, a memory, or an LSI, and implemented by software,such as a program loaded into a memory. Therefore, it will be understoodby those skilled in the art that the functional blocks may be variouslyimplemented by hardware only, by software only, or by a combination ofhardware and software. The method of implementing the functional blocksis not specifically limited.

The image storage device 16 stores a plurality of viewpoint imagesacquired from a plurality of viewpoint positions in association withcoordinate values indicative of viewpoint positions in two-dimensionalcoordinates. The virtual two-dimensional coordinate plane is set foreach of the multi-viewpoint images simultaneously captured by themulti-viewpoint camera 18. In the embodiment, the multi-viewpoint camera18 performs an object imaging operation by changing the angle in thehorizontal plane around a predetermined position by a predeterminedangle of β at a time while maintaining a horizontal gaze direction ofthe multi-viewpoint camera 18. If, for example, the predetermined angleof β is 0.1 degrees, 3600 different multi-viewpoint images are generatedwhen a total of 3600 imaging operations are performed while themulti-viewpoint camera 18 is rotated by 0.1 degrees at a time. Further,if, for example, the predetermined angle of β is 10 degrees, 36different multi-viewpoint images are generated when a total of 36imaging operations are performed while the multi-viewpoint camera 18 isrotated by 10 degrees at a time. Performing such imaging operationscauses the image storage device 16 to store a 360-degree panoramicimage. It should be noted that the smaller the rotation angle β for theimaging operations, the more preferable parallax images can be suppliedto the user.

The sensor information acquisition section 20 acquires the sensorinformation of the posture sensor 124 and supplies the acquired sensorinformation to the motion detection section 30. Further, the capturedimage acquisition section 22 acquires a captured image and supplies theacquired captured image to the motion detection section 30.

The motion detection section 30 detects the motion of the HMD 100 wornon the head of the user, and identifies the position coordinates andposture of the HMD 100. This process is performed to ensure that theviewing field to be displayed on the display panel 130 of the HMD 100coordinates with the motion of the user's head. This head trackingprocess may be performed by using a well-known method.

The motion detection section 30 detects the position coordinates andposture of the HMD 100 from the sensor information of the posture sensor124. The motion detection section 30 identifies the position coordinatesof the HMD 100 from the sensor information of the three-axisacceleration sensor, and identifies the posture of the HMD 100 from thesensor information of the three-axis gyro sensor. The positioncoordinates are derived from a position coordinate in the pitch axisdirection (X-coordinate), a position coordinate in the yaw axisdirection (Y-coordinate), and a position coordinate in the roll axisdirection (Z-coordinate) when the reference position is used as theorigin. Further, the posture of the HMD 100 is derived from an anglearound the pitch axis (pitch angle) with respect to the referenceposture, an angle around the yaw axis (yaw angle) with respect to thereference posture, and an angle around the roll axis (roll angle) withrespect to the reference posture.

It is preferable that the motion detection section 30 enhance theaccuracy of detection of the position coordinates and posture by furtherusing the result of imaging the light-emitting markers 110 for tracking.The motion detection section 30 detects the position coordinates andposture of the HMD 100 at predetermined intervals. If, for example,images are supplied to the HMD 100 at a frame rate of 60 fps, it ispreferable that the motion detection section 30 execute a detectionprocess at 1/60-second intervals. The following describes a method thatis used by the information processing device 10 to generate the parallaximages in accordance with position information and posture regarding theHMD 100.

Based on the position information and posture regarding the HMD 100,which are detected by the motion detection 30, the status determinationsection 32 determines the viewpoint position, gaze direction, andbinocular inclination angle of the user. In the embodiment, the user'sviewpoint position is defined as the position of a central point betweenthe left and right eyes. Further, the user's gaze direction is definedas a direction toward which the user faces with the user's viewpointposition defined as a starting point. Furthermore, the binocularinclination angle is defined as the angle between the horizontal planeand the line connecting the left and right eyes of the user.

The status determination section 32 determines the user's viewpointposition based on the position information regarding the HMD 100, andthen determines the viewpoint position of the virtual camera 8 based onthe determined user's viewpoint position. However, if the viewpointposition of the virtual camera 8 is fixed in the VR application, theviewpoint position of the virtual camera 8 may be set at a fixedposition within the virtual cylindrical body. In this case, theviewpoint position of the virtual camera 8 is set at a predeterminedposition on the central axis line in the virtual cylindrical body (seeFIG. 6).

The status determination section 32 determines the user's gaze directionbased on the posture of the HMD 100. The status determination section 32may directly determine the determined user's gaze direction as the gazedirection (optical axis direction) of the virtual camera 8, or maydetermine the gaze direction of the virtual camera 8 by performing acorrection process. If stable sensor information is not supplied to themotion detection section 30 due, for instance, to noise superimposed onthe sensor information, the motion detection section 30 may erroneouslydetect a swinging motion of the user's head while the user's head is notmoving. In such an instance, the status determination section 32 maysmooth the motion detected by the motion detection section 30 forcorrection purposes, and determine the user's gaze direction and thegaze direction of the virtual camera 8.

In any case, the user's gaze direction corresponds to the gaze direction(optical axis direction) of the virtual camera 8 disposed in the virtualcylindrical body (see FIG. 6). The virtual camera 8 has a defaultbaseline length of 20 mm.

FIGS. 10(a) and 10(b) are diagrams illustrating the binocularinclination angle. It is presumed in FIGS. 10(a) and 10(b) that theuser's gaze direction is parallel to the horizontal plane. The directionof the line connecting the left and right eyes of the user may bedefined as the direction of housing width of the HMD 100.

FIG. 10(a) depicts the posture of the HMD 100 when the binocularinclination angle is zero. In this case, the user's head is notlaterally tilted. Therefore, a line A connecting the left and right eyesof the user is parallel to the horizontal plane.

FIG. 10(b) depicts the posture of the HMD 100 when the binocularinclination angle is α. When the user's head is laterally tilted, anangle is formed between the horizontal plane and the line connecting theleft and right eyes. In the example of FIG. 10(b), the user's head istilted leftward as viewed from the user so that an angle of α is formedbetween the horizontal plane and a line B connecting the left and righteyes. In this example, a binocular inclination angle of α is equivalentto the roll angle.

The following describes a method that is used by the imageidentification section 34 to identify two images for generating theleft- and right-eye parallax images from a plurality of viewpoint imagesstored in the image storage device 16 in accordance with at least theuser's gaze direction and binocular inclination angle determined by thestatus determination section 32.

As described above, the image storage device 16 stores a plurality ofdifferent multi-viewpoint images in association with gaze directions atthe time of acquisition. In a case where the multi-viewpoint images arecaptured by the multi-viewpoint camera 18 at 0.1-degree intervals, 3600different multi-viewpoint images are stored in the image storage device16. Each multi-viewpoint image includes a plurality of viewpoint images,and each viewpoint image is stored in the image storage device 16 inassociation with a viewpoint position in two-dimensional coordinatesthat is set for each multi-viewpoint image.

From a plurality of different multi-viewpoint images acquired indifferent gaze directions, the image identification section 34identifies multi-viewpoint images acquired in a gaze directioncoinciding with the user's gaze direction. If, for example, the user'sgaze direction in the status determination section 32 has a resolutionof 0.1 degrees, and the image storage device 16 stores multi-viewpointimages acquired in a gaze direction spaced at 0.1-degree intervals,multi-viewpoint images acquired in a direction coinciding with theuser's gaze direction are definitely stored. The following describes acase where multi-viewpoint images acquired in a gaze directioncoinciding with the user's gaze direction are the multi-viewpoint imagesdepicted in FIG. 5.

First of all, in a virtual two-dimensional coordinate plane ofmulti-viewpoint images acquired in a gaze direction coinciding with theuser's gaze direction, the image identification section 34 determinescoordinate values such that a gaze direction starting from the user'sviewpoint position penetrates the two-dimensional coordinate plane ofthe multi-viewpoint images. The user's gaze direction is orthogonal tothe two-dimensional coordinate plane.

FIG. 11 illustrates an intersection between the user's gaze directionand the two-dimensional coordinate plane. Coordinate values of theintersection represent the viewpoint position of the virtual camera 8facing the two-dimensional coordinate plane, that is, the user'sviewpoint position. A position represented by the coordinate values ofthe intersection between the two-dimensional coordinate plane and theuser's gaze direction is hereinafter referred to as the user's “gazedirection position EP.”

FIG. 12 illustrates an example in which the user's gaze directionposition EP in the two-dimensional coordinate plane is derived. Here,the user's gaze direction position EP is derived as a positionrepresented by the coordinate values (0,3) in the two-dimensionalcoordinate plane.

When the gaze direction position EP is identified, the imageidentification section 34 identifies two images for generating the left-and right-eye parallax images in accordance with the binocularinclination angle determined by the status determination section 32. Aposition represented by the coordinate values of a viewpoint image usedfor generating the left-eye image 5 a is hereinafter referred to as the“left-eye image position LP,” and a position represented by thecoordinate values of a viewpoint image used for generating the right-eyeimage 5 b is hereinafter referred to as the “right-eye image positionRP.”

FIG. 13(a) illustrates an example of an identified left-eye imageposition LP and an identified right-eye image position RP. FIG. 13(a)depicts the left-eye image position LP and right-eye image position RPthat are identified when the user's head is not tilted laterally asdepicted in FIG. 10(a).

The image identification section 34 identifies two images that arepositioned at the same distance from the gaze direction position EP in adirection along the binocular inclination angle, and are disposed insuch a manner that the distance between the viewpoint positions of thetwo images is equal to a preset distance. That is to say, the imageidentification section 34 determines the left-eye image position LP andthe right-eye image position RP with reference to the gaze directionposition EP in such a manner that the left-eye image position LP, thegaze direction position EP, and the right-eye image position RP arelined up straight, and that the distance from the gaze directionposition EP to the left-eye image position LP is equal to the distancefrom the gaze direction position EP to the right-eye image position RP.

Here, the preset distance between the left-eye image position LP and theright-eye image position RP corresponds to the baseline length of thevirtual camera 8, that is, the distance between the left camera and theright camera. In the embodiment, the virtual camera 8 has a defaultbaseline length of 20 mm. Therefore, the image identification section 34determines the left-eye image position LP and the right-eye imageposition RP in such a manner that the parallax between the left- andright-eye image is 20 mm. In the present example, the binocularinclination angle, which is the angle between the horizontal plane andthe line connecting the left and right eyes of the user, is zero.Therefore, the left-eye image position LP is identified as a positionrepresented by the coordinates (−5,3), and the right-eye image positionRP is identified as a position represented by the coordinates (5,3).

FIG. 13(b) illustrates another example of the identified left-eye imageposition LP and right-eye image position RP. FIG. 13(b) depicts theleft-eye image position LP and right-eye image position RP that areidentified when the user's head is tilted laterally as depicted in FIG.10(b). The example of FIG. 13(b) indicates a case where the user in astate depicted in FIG. 10(a) tilts his/her head leftward as depicted inFIG. 10(b). Therefore, the gaze direction position EP movescounterclockwise from the coordinates (0,3) to the coordinates (−1,2).

The image identification section 34 identifies two images that arepositioned at the same distance from the gaze direction position EP in adirection along the binocular inclination angle, and are disposed insuch a manner that the distance between the viewpoint positions of thetwo images is equal to a preset distance. In the present example, theresult of identification indicates that the binocular inclination angleis α, and that the left-eye image position LP is represented by thecoordinates (−5,−1), and further that the right-eye image position RP isrepresented by the coordinates (3,5).

In the embodiment, the left-eye image position LP and the right-eyeimage position RP are derived by using the following parameters:

Parameters

Gaze direction position EP: (x0,y0)

Preset parallax distance: ipd

Binocular inclination angle: α

Inter-image distance in two-dimensional coordinates: s

The x-direction parallax between the left-eye image position LP and theright-eye image position RP is ipd×cos(α), and the y-direction parallaxbetween the left-eye image position LP and the right-eye image positionRP is ipd×sin(α). Therefore, the following equations can be used forderivation:Coordinates of left-eye image positionLP={(x0−(ipd/s)×cos(α)/2),(y0−(ipd/s)×sin(α)/2)}Coordinates of right-eye imagepositionRP={(x0+(ipd/s)×cos(α)/2),(y0+(ipd/s)×sin(α)/2)}

In the embodiment, ipd=20 mm and s=2 mm. Therefore, the followingexpressions can be used:Coordinates of left-eye image positionLP={x0−5×cos(α),y0−5×sin(α)}Coordinates of right-eye image positionRP={x0+5×cos(α),y0+5×sin(α)}

It is preferable that the preset parallax distance ipd be freelychangeable by the user. Referring to FIG. 9, the reception section 24 isable to receive the preset distance specified by the user. When thereception section 24 receives the specified preset distance, the imageidentification section 34 changes the preset distance in accordance withthe specification received by the reception section 24. A change in thepreset distance is equivalent to a change in the ipd parameter value ofthe derivation equations.

As described above, the image identification section 34 identifiesmulti-viewpoint images serving as materials for parallax images from aplurality of different multi-viewpoint images stored in the imagestorage device 16 in accordance with the user's gaze direction, andfurther identifies two images for use in the generation of parallaximages in accordance with the binocular inclination angle from aplurality of viewpoint images included in the identified multi-viewpointimages. In the example illustrated, for instance, in FIG. 13(a), theimage identification section 34 decides to use a viewpoint imageassociated with the coordinates (−5,3) for the generation of theleft-eye image 5 a and use a viewpoint image associated with thecoordinates (5,3) for the generation of the right-eye image 5 b.Further, in the example illustrated in FIG. 13(b), the imageidentification section 34 decides to use a viewpoint image associatedwith the coordinates (−5,−1) for the generation of the left-eye image 5a and use a viewpoint image associated with the coordinates (3,5) forthe generation of the right-eye image 5 b.

The image generation section 36 generates left- and right-eye parallaximages from the two viewpoint images identified by the imageidentification section 34. In this instance, the image generationsection 36 generates parallax images by adjusting the angles of the twoimages in accordance with the binocular inclination angle.

When the binocular inclination angle is not zero, that is, when theuser's head is tilted laterally, the top-to-bottom direction of theleft-eye display panel 130 a and right-eye display panel 130 b is tiltedfrom the vertical direction. Therefore, when the top-to-bottom directionof the left-eye display panel 130 a and right-eye display panel 130 b isadjusted to coincide with the top-to-bottom direction of a displayedimage, an image tilted by the binocular inclination angle α is visibleto the user's eyes.

Accordingly, the image generation section 36 generates the left-eyeimage 5 a and the right-eye image 5 b by rotating the two viewpointimages identified by the image identification section 34 through anangle of −α and cutting out angle-of-view images of the left-eye displaypanel 130 a and right-eye display panel 130 b from the viewpoint imagesrotated through an angle of −α. That is to say, the image generationsection 36 adjusts the angles of the two viewpoint images so that thetop-to-bottom direction of images displayed on the left-eye displaypanel 130 a and the right-eye display panel 130 b coincides with theactual top-to-bottom direction.

Viewpoint images stored in the image storage device 16 have a largerangle of view than the images to be displayed on the left-eye displaypanel 130 a and the right-eye display panel 130 b. Therefore, theleft-eye image 5 a and right-eye image 5 b corresponding to the anglesof view of the left-eye display panel 130 a and right-eye display panel130 b can be cut out from the viewpoint images rotated through an angleof −α. It should be noted that image light from the display panels 130reaches the user's eyes through the optical lenses. Therefore, it ispreferable that the image generation section 36 generate parallax imagesafter correcting distortion caused by passage through the opticallenses.

When the image generation section 36 generates the parallax images, theimage supply section 38 supplies the parallax images to the HMD 100. Thecontrol section 120 in the HMD 100 causes the left-eye display panel 130a to display a left-eye image and causes the right-eye display panel 130b to display a right-eye image. This enables the user to view theparallax images displayed on the left-eye display panel 130 a and theright-eye display panel 130 b.

As described above, even when the left and right eyes of the user differin horizontal height, the information processing device 10 is able tosupply appropriate parallax images to the HMD 100 by generating parallaximages from a plurality of viewpoint images included in themulti-viewpoint images. In the embodiment, the image storage device 16stores, as the viewpoint images, a plurality of actual viewpoint imagescaptured by the cameras 19 and a plurality of interpolated imagesgenerated from at least two actual viewpoint images by the interpolationprocess. Therefore, the image identification section 34 selects twoimages for use in the generation of parallax images from the actualviewpoint images and the interpolated images (virtual viewpoint images).

In another example, the image storage device 16 stores only the actualviewpoint images captured by the cameras 19 as the viewpoint images. Inthis instance, the image identification section 34 may generate twoimages for use in the generation of the left- and right-eye parallaximages from the actual viewpoint images stored in the image storagedevice 16 by performing a real-time interpolation process based on theuser's gaze direction and binocular inclination angle determined by theimage identification section 34. This provides an advantage in that alarge number of interpolated images need not be generated in advance.

For the real-time interpolation process, the image storage device 16 maystore a plurality of actual viewpoint images and some interpolatedimages. The some interpolated images are used to promptly generatevirtual viewpoint images having desired coordinate values, and may be,for example, interpolated images positioned midway between two actualviewpoint images. When the interpolation process is to be performed inreal time as described above, a way of increasing the speed of theinterpolation process may devised.

The above-described process is performed in a case where the imageidentification section 34 identifies, from a plurality of differentmulti-viewpoint images, multi-viewpoint images captured in the same gazedirection as the user's gaze direction. When the user's gaze directionhas a resolution of 0.1 degrees and the image storage device 16 storesmulti-viewpoint images acquired in a gaze direction spaced at 0.1-degreeintervals, the image identification section 34 is able to identifymulti-viewpoint images captured in the same gaze direction as the user'sgaze direction.

Meanwhile, when the multi-viewpoint images are captured in a gazedirection spaced at small angular intervals, there may be a case whereno multi-viewpoint images match the user's gaze direction. A processperformed in such a case is described below.

FIG. 14 is a diagram illustrating the gaze direction and angle-of-viewof multi-viewpoint images. FIG. 14 illustrates the gaze direction andangle-of-view of actual viewpoint images captured from above by themulti-viewpoint camera 18 and of virtual viewpoint images (interpolatedimages). Individual viewpoint images are obtained by imaging an objectat a predetermined angle of view around individual gaze directions. Itshould be noted that FIG. 14 depicts ten different gaze directions forthe sake of convenience. In reality, however, twenty-one gaze directionsare laterally arrayed as described in conjunction with the embodiment. Atwo-dimensional coordinate plane 15 is a virtual two-dimensional planefor expressing a viewpoint position for each viewpoint image. Thedistance between the multi-viewpoint camera 18 and the two-dimensionalcoordinate plane 15 corresponds to the radius of the virtual cylindricalbody.

FIG. 15 is a diagram illustrating the relationship betweentwo-dimensional coordinate planes 15 of multi-viewpoint images that arecaptured by rotating the multi-viewpoint camera 18 through apredetermined angle of R. Here, the multi-viewpoint camera 18 a capturesan object image in a first gaze direction, and the multi-viewpointcamera 18 b captures the object image in a second gaze direction. Thefirst gaze direction of the multi-viewpoint camera 18 a and the secondgaze direction of the multi-viewpoint camera 18 b differ by an angle ofβ.

In the above-described embodiment, the image identification section 34selects multi-viewpoint images acquired in the same gaze direction asthe user's gaze direction, and then identifies two viewpoint images forforming parallax images from the selected multi-viewpoint images. Asregards the multi-viewpoint images depicted, for example, in FIG. 15,the image identification section 34 selects multi-viewpoint imagescaptured by the multi-viewpoint camera 18 a when the user's gazedirection coincides with the first gaze direction, and selectsmulti-viewpoint images captured by the multi-viewpoint camera 18 b whenthe user's gaze direction coincides with the second gaze direction.

The following describes how the image identification section 34 selectsmulti-viewpoint images in a case where the vector of the user's gazedirection exists between the vector of the first gaze direction and thevector of the second gaze direction. More precisely, the followingdescribes how the image identification section 34 selectsmulti-viewpoint images in a case where the vector of the user's gazedirection is projected onto a plane defined by the vector of the firstgaze direction and the vector of the second gaze direction, and then thedirection of the projected vector is between the first gaze directionand the second gaze direction.

FIG. 16 illustrates an exemplary user's gaze direction S. In FIG. 16,the user's gaze direction S is such that the rotation center of themulti-viewpoint cameras 18 is the viewpoint position (starting point).However, the viewpoint position is not limited to such a position.

In a case where the user's gaze direction S does not coincide with thefirst gaze direction and with the second gaze direction, the imageidentification section 34 identifies at least two images for use in thegeneration of the left- and right-eye parallax images from a pluralityof viewpoint images acquired in the first gaze direction and/or aplurality of viewpoint images acquired in the second gaze direction inaccordance with the user's gaze direction and binocular inclinationangle determined by the status determination section 32.

As an example, the image identification section 34 may compare the angleA1 between the first gaze direction and the user's gaze direction S withthe angle A2 between the second gaze direction and the user's gazedirection S, and select multi-viewpoint images closer in direction tothe user's gaze direction S, that is, multi-viewpoint images having arelatively small angle. If, for example, the angle A1 is larger than theangle A2, the image identification section 34 decides to identify twoimages for use in the generation of parallax images by using a secondimage group including a plurality of parallax images acquired in thesecond gaze direction. If, by contrast, the angle A2 is larger than theangle A1, the image identification section 34 decides to identify twoimages for use in the generation of parallax images by using a firstimage group including a plurality of parallax images acquired in thefirst gaze direction. As described above, based on the user's gazedirection S, the image identification section 34 may decide to identifytwo images by using either the first image group including the viewpointimages acquired in the first gaze direction or the second image groupincluding the viewpoint images acquired in the second gaze direction.

It should be noted that the two-dimensional coordinate plane 15 is notorthogonal to the user's gaze direction S no matter which of the firstand second image groups is used. Therefore, based on the user's gazedirection S, the image generation section 36 needs to perform aprojective transformation on the image group to be used.

As another example, the image identification section 34 may identify atleast two images for use in the generation of parallax images by usingboth the first image group including the viewpoint images acquired inthe first gaze direction and the second image group including theviewpoint images acquired in the second gaze direction.

FIG. 17 is a diagram illustrating an example in which parallax imagesare generated by using both the first image group and the second imagegroup. FIG. 17 depicts the positional relationship between atwo-dimensional coordinate plane 15 a, a two-dimensional coordinateplane 15 b, and the user's gaze direction S. A two-dimensionalcoordinate plane 15 s is a virtual plane orthogonal to the user's gazedirection.

Here, it is assumed that the left-eye image position LP and right-eyeimage position RP in the two-dimensional coordinate plane 15 s areidentified by the image identification section 34. When lines orthogonalto the two-dimensional coordinate plane 15 s are drawn from the left-eyeimage position LP and the right-eye image position RP, and theorthogonal lines intersect the two-dimensional coordinate plane 15 a andthe two-dimensional coordinate plane 15 b, viewpoint images associatedwith the intersections can be used to generate the parallax images.

In the example of FIG. 17, the orthogonal line passing through theleft-eye image position LP intersects the two-dimensional coordinateplane 15 a and the two-dimensional coordinate plane 15 b. Meanwhile, theorthogonal line passing through the right-eye image position RPintersects the two-dimensional coordinate plane 15 b, but does notintersect the two-dimensional coordinate plane 15 a. This signifies thata viewpoint image associated with the right-eye image position RP in thetwo-dimensional coordinate plane 15 s does not exist in thetwo-dimensional coordinate plane 15 a. It should be noted that FIG. 17depicts the relationship between the two-dimensional coordinate planes15 in a one-dimensional manner. In reality, however, the intersectionsare determined in a two-dimensional manner.

The image generation section 36 is capable of combining a viewpointimage acquired in the first gaze direction with a viewpoint imageacquired in the second gaze direction. Before this combination, aprojective transformation is performed on each viewpoint image inaccordance with the user's gaze direction S. In the example of FIG. 17,the image generation section 36 combines a viewpoint image acquired inthe first gaze direction with a viewpoint image acquired in the secondgaze direction for the purpose of generating the left-eye image 5 a. Astwo viewpoint images are used to generate the left-eye image 5 a, imagereproduction can be performed with increased accuracy. For the purposeof generating the right-eye image 5 b, the image generation section 36generates the right-eye image 5 b from a viewpoint image in the secondgaze direction because there is no viewpoint image in the first gazedirection.

The present invention has been described based on an embodiment. It isto be understood by those skilled in the art that the embodiment isillustrative, and that a combination of the elements and processesdescribed in conjunction with the embodiment can be variously modified,and further that such modifications can be made without departing fromthe spirit and scope of the present invention.

The embodiment has been described on the assumption that the user's gazedirection is parallel to the horizontal plane. However, the presentinvention is not limited to such a case. As is the case with a processthat is performed in the embodiment when the user's gaze direction doesnot coincide with the gaze direction of multi-viewpoint images, ifviewpoint images are acquired in a gaze direction parallel to thehorizontal plane and the user's gaze direction is not parallel to thehorizontal plane, the image generation section 36 is able to perform aprojective transformation on the viewpoint images in order to equalizethe gaze direction for image acquisition with the user's gaze directionand generate suitable parallax images.

REFERENCE SIGNS LIST

1 Information processing system, 10 Information processing device, 12Processing device, 14 Output control device, 16 Image storage device, 20Sensor information acquisition section, 22 Captured image acquisitionsection, 24 Reception section; 30 Motion detection section, 32 Statusdetermination section, 34 Image identification section, 36 Imagegeneration section, 38 Image supply section

INDUSTRIAL APPLICABILITY

The present invention is applicable to a technical field where images tobe displayed on a head-mounted display are generated.

The invention claimed is:
 1. An information processing devicecomprising: a detection section that detects the posture of ahead-mounted display worn on the head of a user; a status determinationsection that determines a user's gaze direction and a binocularinclination angle in accordance with the posture of the head-mounteddisplay that is detected by the detection section, the binocularinclination angle being the angle between a horizontal plane and a lineconnecting the left and right eyes of the user; an image identificationsection that identifies two images for use in the generation of left-eyeand right-eye parallax images from a plurality of viewpoint images inaccordance with the user's gaze direction and the binocular inclinationangle that are determined by the status determination section, whereinthe plurality of viewpoint images are arranged in a 3×3 or greaterrectangular grid, and wherein the left-eye parallax image is selectedbased on a distance from a left-eye gaze point to a first gaze pointimage, and wherein the right-eye parallax image is selected based on adistance from a right-eye gaze point to a second image from theplurality of viewpoint images; an image generation section thatgenerates the left-eye and right-eye parallax images from the two imagesidentified by the image identification section; and an image supplysection that supplies the parallax images generated by the imagegeneration section to the head-mounted display.
 2. The informationprocessing device according to claim 1, wherein the image generationsection generates the parallax images by adjusting the angles of the twoimages in accordance with the binocular inclination angle.
 3. Theinformation processing device according to claim 1, wherein theviewpoint images are stored in an image storage device.
 4. Theinformation processing device according to claim 3, wherein the imagestorage device stores each of the viewpoint images in association with aviewpoint position.
 5. The information processing device according toclaim 4, wherein the image identification section identifies two imageswhose viewpoint positions are apart from each other by a presetdistance.
 6. The information processing device according to claim 5,wherein the image identification section identifies a user's gazedirection position in two-dimensional coordinates, and identifies twoimages that are positioned at the same distance from the user's gazedirection position in a direction along the binocular inclination angleand are disposed in such a manner that the distance between theviewpoint positions of the two images is equal to the preset distance.7. The information processing device according to claim 5, furthercomprising: a reception section that receives the preset distancespecified by the user; wherein the image identification section changesthe preset distance in accordance with the specified preset distancereceived by the reception section.
 8. The information processing deviceaccording to claim 3, wherein the image storage device stores, as aplurality of viewpoint images, a plurality of actual viewpoint imagescaptured by a camera and a plurality of interpolated images, theinterpolated images being generated by performing an interpolationprocess on at least two actual viewpoint images to form the rectangulargrid; and wherein the image identification section selects two imagesfor use in the generation of parallax images from the actual viewpointimages and the interpolated images.
 9. The information processing deviceaccording to claim 3, wherein the image storage device stores aplurality of actual viewpoint images captured by the camera; and whereinthe image identification section generates two images for use in thegeneration of left-eye and right-eye parallax images by performing theinterpolation process on the actual viewpoint images stored in the imagestorage device in accordance with the user's gaze direction andbinocular inclination angle determined by the status determinationsection.
 10. An image generation method comprising: detecting theposture of a head-mounted display worn on the head of a user;determining a user's gaze direction and a binocular inclination angle inaccordance with the detected posture of the head-mounted display, thebinocular inclination angle being the angle between a horizontal planeand a line connecting the left and right eyes of the user; identifyingtwo images for use in the generation of left- and right-eye parallaximages from a plurality of viewpoint images in accordance with theuser's gaze direction and the binocular inclination angle, wherein theplurality of viewpoint images are arranged in a 3×3 or greaterrectangular grid, and wherein the left-eye parallax image is selectedbased on a distance from a left-eye gaze point to a first gaze pointimage, and wherein the right-eye parallax image is selected based on adistance from a right-eye gaze point to a second image from theplurality of viewpoint images; generating the left- and right-eyeparallax images from the two identified images; and supplying thegenerated parallax images to the head-mounted display.
 11. Aninformation processing device comprising: a detection section thatdetects the posture of a head-mounted display worn on the head of auser; a status determination section that determines a user's gazedirection and a binocular inclination angle in accordance with theposture of the head-mounted display that is detected by the detectionsection, the binocular inclination angle being the angle between ahorizontal plane and a line connecting the left and right eyes of theuser; an image identification section that identifies at least twoimages for use in the generation of left- and right-eye parallax imagesfrom a plurality of viewpoint images acquired in a first gaze directionand/or a plurality of viewpoint images acquired in a second gazedirection in accordance with the user's gaze direction and the binocularinclination angle that are determined by the status determinationsection, wherein the plurality of viewpoint images are arranged in a 3×3or greater rectangular grid, and wherein the left-eye parallax image isselected based on a distance from a left-eye gaze point to a first gazepoint image, and wherein the right-eye parallax image is selected basedon a distance from a right-eye gaze point to a second image from theplurality of viewpoint images; an image generation section thatgenerates the left- and right-eye parallax images from at least the twoimages identified by the image identification section; and an imagesupply section that supplies the parallax images generated by the imagegeneration section to the head-mounted display.
 12. The informationprocessing device according to claim 11, wherein when the vector of theuser's gaze direction is projected onto a plane defined by the vector ofthe first gaze direction and the vector of the second gaze direction,the direction of the projected vector is between the first gazedirection and the second gaze direction.
 13. The information processingdevice according to claim 11, wherein the image identification sectiondecides, based on the user's gaze direction, to identify the two imagesby using either a first image group or a second image group, the firstimage group including a plurality of viewpoint images acquired in thefirst gaze direction, the second image group including a plurality ofviewpoint images acquired in the second gaze direction.
 14. Theinformation processing device according to claim 11, wherein the imageidentification section identifies at least two images for use in thegeneration of parallax images by using both the first image group andthe second image group, the first image group including a plurality ofviewpoint images acquired in the first gaze direction, the second imagegroup including a plurality of viewpoint images acquired in the secondgaze direction.
 15. The information processing device according to claim14, wherein the image generation section combines a viewpoint imageacquired in the first gaze direction with a viewpoint image acquired inthe second gaze direction.
 16. An image generation method comprising:detecting the posture of a head-mounted display worn on the head of auser; determining a user's gaze direction and a binocular inclinationangle in accordance with the detected posture of the head-mounteddisplay, the binocular inclination angle being the angle between ahorizontal plane and a line connecting the left and right eyes of theuser; identifying at least two images for use in the generation of left-and right-eye parallax images from a plurality of viewpoint imagesacquired in a first gaze direction and/or a plurality of viewpointimages acquired in a second gaze direction in accordance with the user'sgaze direction and the binocular inclination angle, wherein theplurality of viewpoint images are arranged in a 3×3 or greaterrectangular grid, and wherein the left-eye parallax image is selectedbased on a distance from a left-eye gaze point to a first gaze pointimage, and wherein the right-eye parallax image is selected based on adistance from a right-eye gaze point to a second image from theplurality of viewpoint images; and generating the left- and right-eyeparallax images from at least the two identified images.